One important point to raise here is that one reason to use gossipsub and not a custom protocol is that it's a generic message transport - we can expect that the gossipsub networks will cross-pollinate - say between ipfs and eth2 for example - and it's plausible that one might wish to be a participant of both. In fact, it's one of the main advantages of using gossipsub: peer diversity is increased.

The strict policies here somewhat limit that benefit - it creates, in effect, two different networks. The issue could be mitigated by making sure that implementations support different policies per topic - the way to achieve this would be to move some of the validation up to the application layer which easier can handle the per-topic nuances and instead only have a lax mode in gossipsub where:

• iff a field is present, it must be valid for its type
  • seqno must be 8 bytes
  • public keys and peerid's must parse as valid for their type
  • signatures, if present, must validate according to rules that will be lifted to a different PR
• iff the field is not present, gossipsub doesn't care per se - the application might though.
• this is congruent with the fields being optional in the protobuf description of the messages

on top of that policy, applications could then choose to restrict things further within the topic namespace they occupy - ie ETH2 might choose to not propagate messages with gossipsub signatures on eth2 topics (because we already do a mostly equivalent check / validation by verifying a payload signature).

Status is developing a similar protocol on top of gossipsub that also doesn't use the gossipsub signatures (because we negotiate a different scheme which is more appropriate for the status use case - the gossipsub signatures themselves live at the wrong level of abstraction which is why their usefulness is limited, often)

In fact, it's one of the main advantages of using gossipsub: peer diversity is increased.

Is this one of the main benefits? That is a clear benefit of using a cross-pollinated DHT but nodes of different networks are not going to generally subscribe to (or be able to execute validations of) gossipsub topics on an alternate network.

well, I'd say so, yes - we're paying the protobuf tax, the "optional-seqno-and-signature-tax", we namespace the topics etc in order to achieve this interoperability where multiple applications can share the infrastructure - and it's not so much about validation specifically but alternatives that might have different needs but still be related (L2, a future ETC fork, client extensions etc).

If it was only ETH2, a much simpler thing to do would have been to take the research behind gossipsub, even forking it, and run it as a separate libp2p protocol (eth2sub/1.0.0) that would use SSZ and pure content-hashing - content-hashing is already a fairly significant departure from "standard" gossipsub - allowing per-topic optionality in the gossipsub fields in a way achieves the same goal but in a softer, more enabling way - ie it's still possible to create a restricted software where only a certain topic space with specific validations rules are applied, but it's also possible to create a mixed network with different rules applied to separate namespaces, without polluting the libp2p spec with application details.

I think @arnetheduck makes a valid point. If we make these signing policies configurable at the entire gossipsub stack level, it's no longer possible to participate in pubsub topics that adopt different signing policies. That is, you could never have a single node subscribed to both Filecoin (lax) and Eth2 (strict no sign) topics, which does harm interoperability and sabotages the feasibility of very desirable cross-chain applications and use cases.

I'm inclined to not merge this PR and iterate on the implementation submitted in libp2p/go-libp2p-pubsub#359 to push these options down to the topic level. That way, when you join to a topic, you specify the policy that applies to that topic.

@protolambda I know you are probably backlogged. Maybe someone in the libp2p or IPFS teams could take this on.

@raulk pushing these options down to a topic level would be great, sounds like a good solution. And yes, quite backlogged with work. I think it is fair to add a notice in this PR that the options can be implemented on a per-topic basis. Then expand on that once the go-libp2p implementation has decided on the best approach.

@protolambda makes sense. I'll rejig the text, and add an implementation margin note that we can remove once this is pushed down to the topic level in go-libp2p.

While working on this, we also noticed that this breaks the default message id generation, which relies on sequence number and peer id, is somewhat broken when the fields are not included - ie migrating fully to a content-based hash method when the anonymous mode is enabled is necessary.

@arnetheduck yes, good point. And so message ID should be per-topic as well (it's kind of already, if you switch based on the topic info within the message). Will update PR now to incorporate formatting + per-topic behavior recommendation.

@raulk update:

• Implemented your PR suggestions (or at least my interpretation of them, I wanted to clarify and format some more details).
• Added a section about message identification, and clarify the two common flavors, as well as moving the extra details from the fields section there.
• Message signature policy now states that it can initially be a global configuration, but per-topic policies facilitate mixed protocols better
• Optionality of fields is clarified, and states to depend on message ID and signature policy choices.

Again, feel free to make editorial choices, squash/rename commits, or any feedback. I would like to keep this PR moving, so we can continue with ethereum/consensus-specs#2060 before the next Eth2 specs release.

Apologies. Will get to this later today.

I'm going to commit my suggestions, and I'll do another pass tomorrow. Also requesting @vyzo's review here.